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Abstract 

This paper studies the iteration-complexity of new regularized hybrid proximal extragradi¬ 
ent (HPE)-type methods for solving monotone inclusion problems (MIPs). The new (regularized 
HPE-type) methods essentially consist of instances of the standard HPE method applied to reg¬ 
ularizations of the original MIP. It is shown that its pointwise iteration-complexity considerably 
improves the one of the HPE method while approaches (up to a logarithmic factor) the ergodic 
iteration-complexity of the latter method. 
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1 Introduction 

We consider the monotone inclusion problem (MIP) of finding x such that 

0 G B{x) (1) 

where B is a point-to-set maximal monotone operator. One of the most important schemes for 
solving MIPs is the proximal point method (PPM), proposed by Martinet [3] and further developed 
by Rockafellar m- It is an iterative scheme which, in its exact version, generates a sequence {x^} 
according to Xk = {I + XkB)~^Xk-i (where Afc > 0 is a regularization parameter), or equivalently, 
Xk as the unique solution of the MIP: 0 G XkB{x) -|- x — Xk-i- Among other results, Rockafellar [TT] 
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proposed inexact versions of the PPM based on a summable absolute error criterion and subsequently 
Solodov and Svaiter [mis] proposed new inexact variants based on a hybrid proximal extragradient 
(HPE) relative error criterion. In each step, the variants proposed and studied in [T^], namely the 
HPE method, computes A = Afc > 0 and a triple (y,6, e) = {yk,bk,£k) satisfying 

||A6 + y - x|p + 2Ae < - x|p, (2) 

where x = Xk-i is the current iterate, a G [0,1) is a relative error tolerance and denotes the e- 
enlargement [T] of i3. Moreover, instead of choosing y as the next iterate, the HPE method computes 

= Xk by means of an extragradient step x+ = x — Aft. 

The iteration-complexity of the HPE method was established in [5] with regards to the following 
termination criterion in terms of precisions p > 0 and e > 0: find a triple (y, ft, e) such that 

ftGH[^l(y), ||ft||<A £<£• (3) 

Assuming that the sequence of stepsizes {A^} in the HPE method is bounded below by some 
constant A > 0, the pointwise iteration-complexity result of [5] guarantees that the most recent 
triple (y, ft, e) satisfying ([2]) will eventually satisfy the termination criterion given in ([3]) in at most 
O (max {dQ/A^p^, dQ/Ae}) iterations where do denotes the distance of the initial iterate xq to the 
solution set of ([I]). Moreover, under the same condition on the sequence of stepsizes {Afc}, an ergodic 
iteration-complexity result of [5] shows that an ergodic triple constructed from all previous generated 
triples satisfying ([2]) will eventually satisfy ([3]) in at most O (max {do/Ay, d^/Xe}) iterations. Clearly, 
the ergodic iteration-complexity is better than the pointwise one by a factor of O (max{l, do/Ay}). 

Our main goal in this paper is to present regularized HPE-type methods for solving ([1]) which 
essentially consists of instances of the HPE method applied to the regularized MIP 

0 G B{x) + p{x - xo) (4) 


where y > 0 and xq is an initial point. In particular, it is shown that a certain version of the 
regularized HPE method which dynamically adjusts y > 0 solves © in at most 


O 



1 -|- max 





(5) 


iterations. This pointwise iteration-complexity bound considerably improves the one for the usual 
HPE method. Also, note that it differs from the ergodic one for the usual HPE method by only 
a logarithmic factor. Einally, we discuss specific instances of the regularized HPE method which 
are based on Tseng’s modified forward-backward splitting (MEBS) method [15] and Korpelevich’s 
extragradient method [2]. 

Previous most related works. In the context of variational inequalities (Vis), Nemirovski [9] 
has established the ergodic iteration-complexity of an extension of Korpelevich’s method, namely, 
the mirror-prox algorithm, under the assumption that the feasible set of the problem is bounded. 
Nesterov uni proposed a new dual extrapolation algorithm for solving Vis whose termination depends 
on the guess of a ball centered at the initial iterate. Applications of the HPE method to the iteration- 
complexity analysis of several zeroth-order (or, in the context of optimization, first-order) methods 
for solving monotone Vis, MIPs and saddle-point problems were discussed by Monteiro and Svaiter 
in [5] and in the subsequent papers mn- The HPE method was also used to study the iteration- 
complexities of first-order (or, in the context of optimization, second-order) methods for solving 


2 




either a monotone nonlinear equation (see Section 7 of [5]) and, more generally, a monotone VI (see 

m)- 

Organization of the paper. Section [2] contains two subsections. Subsection 12. II presents the nota¬ 
tion as well as some basic concepts about convexity and maximal monotone operators. Subsection l2.2l 
is devoted to the study of a specialization of the HPE method for solving inclusions whose underlying 
operator is written as a sum of a (maximal) monotone and a strongly monotone operator. Section [3] 
presents the main contributions of the paper, namely, the presentation of two new regularized HPE 
methods (a static one and a dynamic one) as well as its complexity analysis. Section [J] discusses 
two specific instances of the dynamic regularized HPE method of Section [3] based on Tseng’s MFBS 
method and Korpelevich’s extragradient method. Finally, the appendix presents the proofs of some 
results in Subsection 12.21 

2 Preliminaries 

This section discusses some preliminary results which will be used throughout the paper. Subsection 
O presents the general notation and some basic concepts about convexity, maximal monotone op¬ 
erators, and related issues. Subsection 12.21 describes a special version of the HPE method introduced 
in [12] for solving monotone inclusions whose underlying operators consist of the sum of a (maximal) 
monotone and a strongly (maximal) monotone operator. 

2.1 Basic concepts and notation 

For t > 0, we let log'''(t) := max{log(t), 0}. Let also X be a finite-dimensional real vector space with 
inner product (•, •) and induced norm || • || := VvTT- Given a set-valued operator H : X ^ X, its 
graph and domain are, respectively, Gr(H) := {(x,u) € X x X : v G A{x)} and Dom(H) := {x € 
X : A[x) / 0}. The inverse of H ; X ^ X is A~^ : X =1 X, A~^{v) := {x : v G A{x)}. The sum 
of two set-valued operators H, H ; X ^ X is defined hy A + B ■. X ^ X, {A + B){x) := {a + b G 
X : a G A(x), b G B{x)}. 

An operator A : X ^ X is /r-strongly monotone if /i > 0 and 

{v — v', x — x') > fj,\\x — x'\/{x,v),{x',v') G Gv{A). (6) 

If /i = 0 in the above inequality, then A is said to be a monotone operator. Moreover, A : X ^ X is 
maximal monotone if it is monotone and maximal in the following sense: if H : X ^ X is monotone 
and Gr(A) C Gt{B), then A = B. The resolvent of a maximal monotone operator A : X ^ X 
with parameter A > 0 is (/ -|- AA)“^. It follows directly from this definition that y = {I + AA)“^a: 
if and only if {x — y)/X G A{y). It is easy to see that if A : X ^ X is //-strongly monotone and 
H : X ^ X is monotone, then the sum A -|- H is also //-strongly monotone. In particular, the sum 
of two monotone operators is also a monotone operator. 

The e-enlargement |Tj of a maximal monotone operator H : X ^ X is defined by : X ^ X, 

B^^\x) ■.= {v G X : {v — v',x — x') > —s, V(x',u') G Gr(H)}. (7) 

The following summarizes some useful properties of B^^\ 

Proposition 2.1. Let A, B : X ^ X be maximal monotone operators. Then, 
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(a) if El < 82, then C for every x E X; 

(b) + (i?)['^l(x) C (^ + i?)[^'+^l(x) for every x € X and e,e' > 0; 

(c) A is monotone if, and only if, A C 

(d) A is maximal monotone if, and only if, A = 

Recall that the e-subdifferential of a proper closed convex function / : X —>• M is defined at 
X E X by def{x) := {u E X : f{x') > /(x) -|- {v,x' — x) — e Vx' E X}. When e = 0, then dfo{x) is 
denoted by df{x) and is called the subdifferential of / at x. The simplest example of subdifferential 
is given by considering indicator functions of closed convex sets. Given a closed convex set X C X 
its indicator function is denoted by and is defined hy 5x{x) := 0 if x E X and dx(x) := oo 
otherwise. The normal cone of X is defined by Nx ■= d6x- We also define the projection on X by 
Px ■■= (/ + Nx)~ 

2.2 Solving inclusions with strongly monotone operators 

In this subsection, we consider the MIP 

0 E X(x) -b B{x) (8) 

where the following assumptions hold: 

A.l) X:X^Xisa /U-strongly maximal monotone operator for some /i > 0 (see ([6])); 

A.2) R : X ^ X is maximal monotone; 

A.3) the solution set of ([8]), i.e., {A + R)“^(0), is nonempty. 

We next state a specialized HPE method for solving ([8|) under the assumptions stated above. 
It will be used later on in Section [3] to describe regularized HPE methods for general MIPs whose 
pointwise iteration-complexities improve the ones for the usual HPE method (see IS])- 


Algorithm 1: A specialized HPE method for solving strongly MIPs 
(0) Let xo E X and u E [0,1) be given and set /c = 1; 

(1) choose Afc > 0 and find yk,Vk E X, cjfc E [0,cr], and > 0 such that 

Vk ^ MVk) + B'^^^\yk), \\\kVk + yk-Xk-if+ ‘2.\kek<cFl\\yk-Xk-i\\^] ( 9 ) 

(2) set 

Xk = Xk-i - XkVk, ( 10 ) 

let /c •(— A: -b 1 and go to step I. 

end 

We now make some remarks about Algorithm 1. Eirst, it can be easily checked that if a = 0 then 
Algorithm 1 reduces to the exact proximal point method (PPM) for solving ([8]), i.e., 

Xk = {Xk{A +B) + I)~^Xk-i VA: > 1. 
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Second, since A{y) + C (^ + B)'^^\y) for every y in view of Proposition 12.If bi. it follows that 

Algorithm 1 is a special instance of the HPE method studied in [5]. Third, like in the HPE method, 
step 1 of Algorithm 1 does not specify how to compute the stepsize and the triple (yfc,Ufc,efc). 
Their computation will depend on the instance of the method under consideration. 

The next result derives convergence rates for the sequences {vk} and {sk} generated by Algo¬ 
rithm 1 under the assumption that the sequence of stepsizes {A^} is bounded away from zero. Its 
proof is given in Appendix [Al 

Proposition 2.2. Let do denote the distanee of xq to the solution set of ([8]) and define 

Assume that Afc > A > 0 for every k > 1. Then, for every k > 1, Vk ^ A{yk) + B^^i^^yk), 


1 + (T 
1 — (T 


^ ) 


do, 


£k < 


CJ 


2(1-a2) 


(1-^) 

A 


k-l 


dl 


X* - Xfcll < (1 - e)^/^\\x* - xoll Vx* G (A + B)-^(O). 


3 Regularized HPE methods for solving MIPs 

This section presents regularized HPE-type methods for solving MIPs whose pointwise iteration- 
complexity is superior to the one for the original HPE method (see M)- It is shown that the new 
pointwise bound is worse than the ergodic one for the original HPE method by only a logarithmic 
factor. 

This section considers the MIP © where B : X ^ X is a point-to-set maximal monotone 
operator such that B~^{0) ^ 0, and discusses regularized HPE-type methods which, for a given 
point xo G X, consist of solving MIPs parametrized by a scalar /i > 0 as in (j4]). Observe that ([4]) is 
a regularized version of ©• Its operator is /r-strongly monotone and approaches the one of ([I|) as 
/r > 0 approaches zero. Clearly, (j4]) is a special case of © with A{x) = fj,{x — xq) and its solution 
set is a singleton by Minty’s theorem [1]. 

We denote the distance of xq to the solution sets of © and © by do and respectively. Clearly, 

dfi = \\x*^ - xoW (12) 

where x* denotes the unique solution of ©, i.e., x^. = ih ^B + I) ^(xo). 

The following simple technical result relates d^ with do- 

Lemma 3.1. For every fJ- > 0, d^ < do- 

Proof. Let x* be the projection of xq onto H“^(0). Since 0 G B{x*) and fi{xo — x*) G B(x*), the 
monotonicity of B and the fact that fv > 0 imply that (x* — x*,x* — xq) > 0. Therefore, 


d^ = 


I* ii2 II* *ii2io/* * * \iii* ii2\ii* 

|x — Xoll = Ip ~ Xn\\ -|- 2(x — X„, X„ — Xo) -|- ||x„ — Xoll > Ip —X 


\^ + dl 


and the conclusion follows. 


□ 
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We now state a /U-regularized HPE method for solving ([T]) which is simply Algorithm 1 (with 
A(-) = — xq)) applied to MIP Q but with a termination criterion added. 


Algorithm 2: A static //-regularized HPE method for solving ([T]). 

Input: {xo,a, /t, /?, e) G A x [0,1) x M++ x M++ x M++; 

(0) set A: = 1; 

(1) choose Afc > 0 and find (j/fc, 6^,6^) G A x A x R+ such that 

bk G ||Afc [bk + y{yk - a^o)] + ?/fc - + 2Afcefc < a'^\\yk - Xfc_i|p; (13) 

(2) if \\bk + y{yk - a:o)|| > p ov Sk > e, then set 

Xk = Xk-i - Xk [bk + piVk - a:o)] , (14) 

and /c •(— A; + 1, and go to step 1; otherwise, stop the algorithm and output {yk,bk,Sk)- 

end 


We now make some remarks about Algorithm 2. First, it is the special case of Algorithm 1 in 
which A(-) = //(• — xo), and hence solves the MIP ([4]). Second, since Subsection 12.21 only deals with 
convergence rate bounds, a stopping criterion was not added to Algorithm 1. In contrast. Algorithm 2 
incorporates a stopping criterion (see step 2 above) based on which its iteration-complexity bound 
is derived in Proposition 13.21 and Theorem 13.31 below. Third, it is shown in Theorem 13.31( b) that 
Algorithm 2 solves MIP ([1]) if // is chosen sufficiently small. 


Proposition 3.2. Assume that A^ > A > 0 for all k >1 and let be as in m- Then, Algorithm 
2 with input {xo,a, p, p,e) terminates in at most 


1 1 


2Xp 1 — cr^ 


2 -|- max < log^ 


1 -I- u 


1 — CT 


A. 

J X^p2 


,log^ 




2(1 — a^)Xe 


(15) 


iterations with a triple {yk,bk,ek) which, in addition to satisfying the stopping criterion in step 2 of 
Algorithm 2, namely, 

\\bk +p{yk- xq)\\ < p, £k<e, (16) 


it also satisfies the inequalities 


~ a^oll < I 1 + 






dn ^ 1 1 + 


\\bk\\ < P + // ( 1 + 






VT^ 


dn<P + p{l + 




do, 


VT^ 




(17) 

(18) 


Proof. To prove (I15j) assume that Algorithm 2 has not terminated at the A:-th iteration, and define 
Vk = bk + pivk — xo)- Then, either ||ufc|| > p or Ek > £■ Assume first that ||ufc|| > p. Since Algorithm 
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2 is a special case of Algorithm 1 applied to MIP dH with A{x) = fi{x — xq) and Vk as above, the 
latter assumption and Corollary 12.21 imply that 


P < ll^fcll < 


l + a ( 


I — a 


where 0 is defined in (|lip . Rearranging this inequality, taking logarithms of both sides of the resulting 
inequality and using the fact that log(l — 9) < —9, we conclude that 


k < 1 + 9 ^ log 


1 + cr 

dl \ 

1 — a 

aV/ 


If, on the other hand, £k > e, we conclude by using a similar reasoning that 


k <1 + 9 ^ log 


cr 


-dl 


2(1 - o-2)Ae 


From the above two observations and the fact that 0 < 1 in view of dill) . (|15p follows. 
To prove (flTl) . note that Lemma 2.1(5) of [14], Corollary 12.21 and (fT^ imply that 


Vi — 






and hence that 


hk - a;o|| < Wvk - 2;»|| + ||3;» - a^oll < 1 + 


Vl — 


The latter conclusion and Lemma l3 .1 1 yield dl7ll . To hnish the proof, note that (|18ll follows from the 
first inequality in (flOp . the triangle inequality and (fT7p . □ 


The complexity results presented in this paper will consist in establishing bounds in the number 
of iterations to obtain a triple (y, b, e) satisfying ([3|), for given precisions p > 0 and e > 0. 

The following result shows that Algorithm 2 solves the MIP ([T|) when // > 0 is chosen sufficiently 
small. 


Theorem 3.3. Assume that Aj, > A > 0 for all k >1 and let a tolerance pair {p, e) € M++ X M++ 
he given. Then, the following statements hold: 


(a) for any p G (0, p) and Vq > 0, Algorithm 2 with input {xo,a, p, p,e) where 

p-p 


p = p{Vq,p) := 


1 + 


1 


\/l — (t2 


e = £ 


(19) 


Pn 


terminates in at most 


1 + i/Vi — <72 


P, 


2A(p - p) 
iterations; 


0 1 

+ 


1-ct2 


2 + max < log 


1 + cr 

1 — C7 


4 


,log 


£72^2 


A2p2y ’ \2(l-f72)Ae 


( 20 ) 
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(b) ifVo > do, then Algorithm 2 with the above input terminates with a triple {yk,bk,£k) satisfying 


G||6fc||<p, Sk<e, y\\yk - xo\\ < p - p. (21) 


Proof. Note that (|20p follows from (jlSp . (I19p and Lemma l3.II Using the second inequalities in (jl7p 
and (fTsp and the first identity in (fT^ we find 

max{||6fc|| - p, pWvk - xo\\} < ^{p-p)- 

L’o 

Thus, if T>o'> do, then the latter inequality yields the second and the fourth inequalities in ([2Tp . The 
inclusion and the third inequality in (j21l) follow from (I18p and (jl6p . respectively. □ 

We now make two remarks about Theorem 13.31 First, if u G [0,1) is such that (1 — = 0(1), 

an upper bound Vo > do such that Vo = O{do) is known, and p is set to p/2, then the complexity 
bound (|20P is 


O 



1 + max 





( 22 ) 


Second, in general an upper bound Pq in the first remark is not known and in such case the bound 
(j20p can be much worse than the one above when Vo » do- 

In the remaining part of this section, we consider the case where an upper bound Pq ^ do such that 
Pq = O{do) is not known and describe a scheme based on Algorithm 2 whose iteration-complexity 
order is equal to ([22]). 


DR-HPE: A dynamic regularized HPE method for solving ([T|). 


(0) Let xo & X, a G [0,1), A > 0 and a tolerance pair (p, e) G IR++ x M_|_+ be given and choose 
p G (0,p); set 


T^o = TAo '■= 


_ 2A(p - p) _, 

(1 - 0-2) ^1 + 1/Vl - 0-2^ 


(23) 


(1) set p = p(Po,p) where p(-,-) is dehned in (fT9P and call Algorithm 2 with input {xo,a, p, p,e) 
to obtain as output {y,b,s)‘, 

(2) if p\\y — xoll < p — p then stop and output {y, b, e); else, set Pq g- 2Po and go to step 1. 

end 


Each iteration of DR-HPE (referred to as an outer iteration) invokes Algorithm 2, and hence 
performs a certain number of iterations of the latter method (called inner iterations) which is bounded 
by (|2njl . The following result gives the overall inner-iteration-complexity of DR-HPE in terms of do, 
A, p, p and e. 







Theorem 3.4. Let do denote the distance of xq to the solution set of © and assume that the 
proximal step size in every inner iteration of DR-HPE is bounded below by a constant A > 0. Then, 
DR-HPE with input {xo,cr,X,{p,£), p) E X x [0,1) x x K^_,_ x such that p E (0, p) and 
(1 — (j)~^ = 0(1) finds a triple {y,b,e) satisfying 


beB^^\y), ||6||<p, e<e 


in at most 


O 



do 


\{p - p) 



1 + max 





iterations. 


(24) 


Proof. Note that at the A:-th outer iteration of DR-HPE, we have Pq = Moreover, in view of 

Theorem I3.3l fbi. DR-HPE terminates in at most K outer iterations where K is the smallest integer 
k>l satisfying 2^~^Vo > do, i.e., 


K = 1 + 



'Do) 


Define 


fii := 


/3o:= 


2 -|- max 



( 

1 + (T 

V 

1 — fJ 


dp \ 

avJ 


,log+ 


ft (l + l/VT^)Po ^ 

1 - 0-2 2A(p — p) ^ 


( ^^^0 A 1 

\2{\-a'^)\e)] 


(25) 

(26) 


where the identity in (I26p follows from (1231) . In view of Theorem I3.3r ai and relations (|25p . (|26ll . we 
then conclude that the overall number of inner iterations of DR-HPE is bounded by 


K 


A. 




i^ + ^(2^-l) 

A 


< fio 






(27) 


To prove the theorem, it suffices to show that K is bounded by (|24l) . Indeed, we consider two 
cases, namely, whether K = 1 oi K > 1. If RT = 1, then (|27ll implies that K < 2/3o(l -|- A/A), 
and hence that the order of K is bounded by (I24p in view of the definition of fio in (12611 . Assume 
now that RT > 1 and note that the definition of K implies that k = K — 1 violates the inequality 
2*^“^Po > do, and hence that 2^ < Ado/Do. The latter conclusion and inequality ((271) then imply 
that K < 4/3odo(l + ^/}i)/Do, which together with (f25|) and (l26p then imply that K is bounded by 
dSlI). □ 

Note that if the lower bound A > 0 for the sequence of proximal stepsizes is known, and A = A 
and p = pI2 are chosen as input for DR-HPE, then the iteration-complexity bound (j24p reduces to 
bound (j22p . This observation justifies our claim preceding DR-HPE. 
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4 Specific instances of the DR-HPE method 

In this section, we briefly discuss speciflc ways of implementing step 1 of Algorithm 2. 
More specifically, we assume that operator B has the structure 


B{x) := F{x) + C{x) 


(28) 


where the following conditions hold: 

B.l) F : Dom(i^) C A —)• A is a (single-valued) monotone operator on Dom(C') C Dom(A), i.e., 

{F{x) — F{x'),x — x') > 0, Vx, x'G Dom(C'); (29) 

B.2) F is L-Lipschitz continuous on a closed convex set hi such that Dom(C) C C Dom(A), i.e., 
there exists L > 0 such that 

||F(x) — F(x')|| < L||x — x'll Vx,x'Gn; (30) 

B.3) C : A ^ A is maximal monotone. 

Our goal in this section is to discuss a Tseng’s modified forward-backward splitting (MFBS) type 
scheme for implementing step 1 of Algorithm 2 for an operator B with the above structure where 
two evaluations of F and a single resolvent evaluation of C, i.e., an operator of the form (/ -|- \C)~^ 
for some A > 0, are made. 

Let {xo,(T,fi) be the first three entities of the input for Algorithm 2 and assume here that a G 
(0,1). Consider the MIP 

0 G B^(x) := F{x) + C'^(x) (31) 

where : A ^ A is defined as 

Cfj^{x) := C{x) + fj,{x — xq) Vx G Dom(C'). (32) 

Given x^-i G A, the following two relations describes an iteration of a variant of Tseng’s MFBS 
algorithm studied in [6] (see also [5]) for the above MIP: 


yk = {I + XC^) ^ (xfc_i - AF(Pn(xfc_i))), (33) 

Xk = Vk - X{F{yk) - F{Pn{xk-i))) (34) 


where A := ajL. Since by assumption B.2 we have Dom(C') C H C Dom(F’), and Dom(C'^) = 
Dom(C), it follows that Po(xfc-i) and yk belong to Dom(F), and hence that the iteration defined 
in (j33p ~ (j34p is well-defined. Moreover, the assumption that the resolvent of C is computable makes 
the resolvent (I -|- AC^)“^ also computable since 


(/ + AC^)-'x = 



A 

1 -|- A/r 



X -|- XyxoX 
1 -|- Xfi ) 


X G A. 


The following proposition was essentially proved in [6l Proposition 4.5] with a different notation. 
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Proposition 4.1. The points yk and Xk defined by (|33p and (|34l) with X = ajL and the vector 

Ck = ^{xk-i - Vk) - F{Pn{xk-i)) - p{yk - xo) 

satisfy 


Ck£C{yk), \\X[F{yk) + Ck+fa{yk-xo)]+yk-Xk-i\\<(7\\yk-Xk-i\\, (35) 

and hence bk '■= F{yifi) + Ck, Xk '■= ^ and Ek := 0 satisfy (fTBI) . 

Proof. The inclusion in (1351) follows directly from (1331) . (|32p and the definition of Ck- On the other 
hand, using items (a) and (c) of [U1 Proposition 4.5] (with a different notation), we obtain the 
inequality in (f35P . The last statement of the proposition follows from the definition of bk , (l28P , (l3^ 
and Proposition 12. If dl. □ 

In the next theorem we show the iteration-complexity of DR-HPE for solving (|28p under the 
assumption that the iteration of the variant of Tseng’s MFBS method described in (f33p ~ (f34p is used 
as an implementation of step 1 of Algorithm 2. 

Theorem 4.2. If max{(T“^,(l — (t)“^} = 0(1), then DR-HPE in whieh step 1 of Algorithm 2 
is implemented according to the reeipe deseribed in Proposition HD terminates with a pair (y, b) 
satisfying 


in at most 


bG{F + C){y), \\b\\<p 


O 



1 + log+ 



(36) 


(37) 


iterations where p and p are as in step 0 of DR-HPE . 

Proof. The result is a direct consequence of Theorem 13.41 and Proposition 14.11 where A = A = A := 
cr/L. We note that since by Proposition 14.11 we have = 0 for all fc > 1 the complexity bound on 
(p^ is independent of the precision e > 0. □ 


We now make some comments about the special instance of DR-HPE described in Theorem 14.21 
in light of a previous variant of Tseng’s MFBS algorithm studied in [6] for solving MIP (I28p . First, 
the cost of an inner iteration of the above two methods are identical. Second, if p = p/2, then the 
complexity bound (f37|) reduces to 


O 




1 -h log+ 



(38) 


which improves the pointwise iteration-complexity bound O ((Ld^/p)"^^ for the variant of Tseng’s 
MFBS algorithm (see [U Theorem 4.6]). Third, it is proved in [5l Theorem 6.2(b)] that the Tseng’s 
MFBS variant finds an ergodic pair (h,y) satisfying b ^ (F Cy(y), ||y|| < p and e < e in at most 
O (max [Ldo/p, Tdg/e]) iterations. Note that the dependence of the latter bound on p differs from 
the one in (j38p only by a logarithmic term. Moreover, in contrast to the latter bound, (|38jl does not 
depend on e. Also, the error criterion implied by the latter ergodic result is weaker than the one 
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in ()36l) . In summary, Theorem 4.2 establishes a pointwise iteration-complexity bound which closely 
approaches the latter ergodic bound while guaranteeing at the same time an error criterion stronger 
than the one for the aforementioned ergodic result. 

We hnish this section by noting that, \iC = dg where g : X ^ {—oo, oo] is a proper closed convex 
function, then an iteration of Korpelevich’s extragradient algorithm (see for example Section 4 of 
m) can also be used to implement step 1 of Algorithm 2 and, as a consequence, yields a different 
instance of DR-HPE. Clearly, it is possible to derive a result for the new variant similar to Theorem 
I4.2l in which the error criterion becomes 6 G (T + dsg){y), ||6|| < p, £ < £ and the complexity bound is 
given by (j24h (and hence depends on e) with A = A = A := ajL. Note that the latter error criterion, 
while weaker than the one in (|36l) . is still stronger than the one of the ergodic result for the Tseng’s 
MBPS variant (see, for instance, [Sj Corollary 5.3(b)]). 

A Proof of Proposition 12.2 

From now on {xk}, {Uk}, {vk}, and {£k} are sequences generated by Algorithm 1. 

Dehne, for A: > 1: 

7fc : X ^ M, 7fc(x) := {vk,x - pk) - £k Vx G X (39) 

and 

Proposition A.l. Let 7 fc(-) and 9k be as in (|3^ and (flOll . respectively. For every k > 1: 

(a) _Xk = argminAfe 7 fc(x) -h ||x - Xfc_i|p/2; 

(b) minAfc 7 fc(x) -|- \\x - Xfc_i|p/2 > (1 - cT^)||yfc - Xfc-i|p/2; 

(c) ik{x*) < -p\\x* - yfclp for any x* ^{A + B)-^(O); 

(d) for any x* G (A -|- R)“^(0), 

||x* - Xk-if > 2\kp\\x* - ykf + (1 - cr‘^)\\yk - Xk-if + \\x* - Xkf 

and 

(1 - 9k) ||x* - Xfc_i|p > ||x* - Xfcll^. 

Proof, (a) This statement follows trivially from (|10p and ()39p . 

(b) Direct use of (fTUP and (f3^ yields, after trivial algebraic manipulations, 

hlk{xk) + ^\\xk - Xk-i\\‘^ = ^ [\\yk - Xk-i\\^ - (IIAfcXfc + yk- Xk-i\\^ + 2Xk£k)] , 

which, combined with item (a) and Q proves item (b). 

(c) If X* G (A -|- B)~^{0), then there exists a* G A(x*) such that —a* G B{x*). It follows from 
the inclusion in ([9]) that there exists Ok G A{yk), bk G B^^i^^yk) such that Vk = Ok + bk. It follows 
from these inclusions, assumption A.l, and ([7]) that 

{a* -ak,x* -yk) > p\\x* - ykf, {bk + a*,yk - x*) > -£k . 
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To end the proof of item (c), add these inequalities, observe that Uk + hk = Vk, and use the definition 
(1391). 

(d) It follows from (f5U|l . (a), and (b) that, for all x G X 

Xklk{x) + ^\\x - Xk-if = ^minAfc7fc(x) + ^||x - + ^||x - 

> ^‘^)\\yk - Xk-lf + Ik - Xkf) ■ 

To prove the first inequality in item (d) take x = x* G (X + i?)“^(0) in the above equation and use 
item (c). To prove the second inequality, observe that ||x* — yk\\ + \\yk — a^fc-ill > Ik* — Xk-i\\, 


min{(l — cT^)r^ + 2/iAfcS^ | r, s > 0, r + s > |k* — a^fc-i||} = ^fclk* “ Xk-i\ 
and use the first inequality of item (d). 


□ 


The following Lemma follows trivially from the inequality in ([9]) , the use of the triangle inequality 
and the fact that > 0. 

Lemma A.2. For k > 1: 

(1 - o-fc) \\yk - Xk-i\\ < IIAfcUfcll < (1 + Ufc) \\yk - Xfc_i||. 

In the next proposition, we establish rates of convergence for the sequences {xk}, {ufc} and {efc} 
generated by Algorithm 1. 


Proposition A.3. Let do denote the distance xq to the solution set of 
k > 1: 


Tk := 


1 1/2 


Ifli-Oj 

J=1 


and define for every 


(41) 


Then, for every fe > 1, Ufc G A{yk) + B^^'^\yk) and 


a 


2(1 - fj2 


£k 

Afc 


dl 


(42) 


k* - xk\\ < Tfcik* - xol 


Vx* G (A + 5)-k0). 


(43) 


Proof. First note that (143 1) follows from the second inequality in Proposition lA.li d) and (j41|) . Using 
the first inequality in Proposition lA.lI di and (Hdll . we conclude that, for all x* G (A + 

(1 - cr^)||yfc - Xk-if < rLilk* - Xof V/c > 1. 

Note now that (j42h follows from the latter inequality and the relations 


£k < 


a 


^k—1 \ 


kfcll < 


{l + a)\\yk - Xk-i\ 


2Afc Xk 

which are due to (l9|) and the second inequality in Lemma lA.21 


□ 
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Proof of Proposition 12.21 The assumption > A > 0 for every k > \ and the fact that the 
scalar function 

is nondecreasing, combined with (|40p and m, imply that 6k > 0 for all /c > 1. From the latter 
inequality and m we obtain F^ < (1 — 6)^^'^ for every k > 1, which, in turn, combined with 
Proposition IA.3I completes the proof. 
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